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SUMMARY 


i  I 


It  Is  becoming  Increasingly  obvious  that  computerized  automation  can  be  a  useful  aid  for  a 
ulde  variety  of  positions  In  the  command  and  control  network  where  many  of  the  tasks  Involve 
situational  assessment  or  "diagnostic  Inference."  To  optimally  combine  human  talent  and 
computer-aiding  systems,  one  must  know  how  the  human  operator  performs  the  task  unaided  (and 
under  what  circumstances),  what  subtasks  can  be  allocated  to  the  machine,  and  what  variables 
affect  operator  acceptance  of  the  aiding  system.  This  paper  presents  a  theoretical  model  of  the 
human  performance  of  a  diagnostic  Inference  task  when  unaided  by  machine.  Including  the  variables 
affecting  those  Inference  processes;  and  a  preliminary  model  of  how  a  computer-aiding  system 
might  be  expected  to  fit  Into  the  diagnostic  system. 
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PREFACE 


The  objective  of  this  research  was  a  theoretical  Model  of  hoe  human  subjects 
perform  a  diagnostic  Inference  task  analdod  by  automation.  In  addition,  a  pro! Ininary 
model  of  how  a  conputer -aiding  device  Might  be  expected  to  fit  Into  the  diagnostic 
system  was  conceptualized. 

This  study  supported  research  being  conducted  In  Ml  SO 17 -06-0*,  Modelling  Impacts 
of  Automation  on  Ron-Automated  Tactical  Command  and  Control  (C2)  Systems.  This 
project  concerns  the  prediction  of  changes  In  cognitive  performance  as  a  function  of 
various  kinds  of  automation.  The  mnthodology  to  be  developed  will  assist  planners  In 
designing  future  automated  systems  that  will  optimize  human  performance.  Technical 
issues  confronting  this  research  concern  the  selection  or  development  of  quantitative 
models  that  can  accurately  depict  human  cognitive  process  and  performance,  and  the 
tools  and  techniques  which  can  captura  tha  higher-level  Interests  between  operator  and 
automation. 

Tha  results  of  the  study  described  In  this  paper  included  a  list  of  variables 
anticipated  to  affect  human  performance  In  an  Inference  task  and  the  Identification  of 
a  candidate  technique  that  can  be  used  to  measure  the  effects  that  these  variables  nay 
have  on  the  Inference  task. 

The  author  would  like  to  thank  the  Air  Force  Systems  Coanand,  the  Air  Force  Office 
of  Scientific  Research,  and  the  Southeastarn  renter  for  Electrical  Engineering 
Education  for  making  possible  a  very  Interesting  and  rewarding  Sunmer  Fellowship  at  the 
Air  Force  Hunan  Resources  Laboratory,  Hr Ight -Patter son  AFB,  Ohio.  She  would  like  to 
express  sincere  appreciation  to  the  Laboratory,  In  particular  the  Bround  Operations 
branch,  for  an  exceptional  working  environment.  Finally,  she  would  like  to  thank 
Rosemarie  Preld;,  for  her  enormous  support  and  collaboration  in  a  difficult 
undertaking,  uertram  Cream  for  his  Insightful  administrative  guidance,  Larry  Reed  for 
many  helpful  discussions,  and  the  other  members  of  the  branch  for  their  professional 
support  and  friendship. 
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MANUAL  AND  COMPUTER-AIDED  SEQUENTIAL 
DIAGNOSTIC  INFERENCE 


I.  INTRODUCTION 

I*  is  oiling  Increasingly  obvious  that  computerized  autonatlon  can  be  a  useful  aid  In  a 
wide  variety  of  positions  In  the  araed  services.  This  Is  especially  true  In  the  world  of  Coaaand 
and  Control  (C2)  where  auch  of  the  work  Involves  complex  situational  assessaent  or  'diagnostic 
Inference.*  As  technological  complexities  Increase,  huaan  operators  will  have  a  wore  difficult 
tlae  trying  to  understand.  Integrate,  and  utilize  the  Inforaatlon  aade  available  to  thea.  In 
contrast  to  aan's  Halted  cognitive  capacities  and  well-docuaented  biases,1*2  a  coaputer  can 
utilize  and  aggregate  large  voluaes  of  Inforaatlon  using  predeteralned  optlaal  strategies  that 
are  aost  appropriate  for  the  situation  at  hand.  It  is  no  longer  a  question  of  whether  coaputer 
aiding  will  be  used,  but  how  It  will  be  used. 

Just  as  there  are  probleas  Inherent  In  using  a  completely  'annual *  systea  to  perfora  a 
function,  there  are  also  probleas  In  using  a  coapletely  'autoaated*  systea.  These  probleas  have 
been  discussed  at  length  elsewhere;2*3  but  let  it  suffice  to  say  that  at  the  current  tlae, 
expert  systems  are  not  sufficiently  advanced  to  Bake  autoaated  systeas  Infallible  or  able  to  deal 
with  the  multitude  of  unforeseen  occurrences  that  are  likely  In  the  C2  environment. 

Since  neither  man  nor  machine  is  solely  capable  of  performing  situational  assessaent 
functions,  the  solution  lies  In  using  both  together  and  relying  on  the  strengths  of  each 
{hopefully  also  minimizing  the  weaknesses  of  each).  To  Integrate  man  and  machine  successfully 
for  a  given  task,  one  must  understand  how  the  huaan  perceives  and  performs  the  task,  and  how  the 
machine  can  be  programmed  to  perform  the  task  (or  parts  of  the  task),  and  then  analyze  the  best 
way  to  fit  the  two  together.  In  the  procurement  cycle,  a  coaaon  method  for  developing  a 
computerized  aiding  system  is  to  Intuitively  develop  a  software  systea  that  seems  as  though  It 
could  do  the  Job.  Little  attention  Is  paid  to  analysis  of  the  entire  task  and  which  subtasks 
could  be  best  performed  by  the  man  and  which  are  best  left  to  the  machine  (a  few  exceptions  do 
exist).  Consideration  Is  usually  not  given  beforehand  to  how  the  operators  will  react  to  the 
aiding  systea  nor  to  what  variables  will  lead  to  their  acceptance  or  rejection  of  the  new 
system.  Instead,  an  autoaated  systea  Is  designed  and  a  prototype  built.  Any  modifications 
necessary  to  make  the  systea  compatible  with  the  operator  are  usually  done  after  this  point. 
This  leads  to  only  those  changes  that  seea  absolutely  necessary  and  the  result  Is  an  overall 
man-machine  system  that  Is  auch  less  effective  and  efficient  than  what  could  have  been  achieved. 

Part  of  the  problem  outlined  above  results  from  an  inadequate  knowledge  concerning  three 
vital  questions:  (a)  How  does  the  huaan  operator  perfora  the  assessaent  task  when  unaided  by 
automation?  (b)  What  subtasks  are  best  performed  by  the  operator,  and  what  subtasks  are  best 
performed  by  the  automation  device?  (c)  What  factors  determine  operator  acceptance  and  use  of 
the  automated  systea?  The  first  question  (how  the  operator  performs  the  task)  aay  seea 
unnecessary  to  soae.  However,  this  Information  Is  needed  because  It  directly  affects  the  answers 
to  the  second  and  third  questions.  That  Is,  If  we  know  how  the  operator  performs  the  task  (not 
how  his/her  performance  differs  from  some  theoretically  optimal  strategy),  we  can  determine 
specifically  what  capabilities  he/she  has  that  we  want  to  preserve  In  determining  the  optimal 
man-machine  subtask  allocation.  In  addition,  one  can  argue  that  how  the  operator  performs  the 
task  will  largely  affect  his/her  acceptance  of  the  automation.  If  the  automation  is  extremely 
different  froa  or  Incompatible  with  the  operator’s  way  of  perceiving  and  accomplishing  the  task, 
then  he/she  will  be  less  likely  to  accept  and  use  that  automation. 

The  theories  and  methodologies  of  cognitive  psychology  can  be  brought  to  bear  on  this 
problem.  By  mapping  owt  the  cognitive  processes  or  strategies  that  are  used  by  the  perceiver 
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under  various  situational  constraints,  we  can  then  Mature  how  those  processes  change  as  a 
function  of  providing  a  computer-aiding  system. 


II.  OBJECTIVES 

The  prlMry  objective  of  this  effort  was  to  develop  a  predictive  model  of  a  diagnostic 

Inference  task  and  how  that  task  would  be  affected  by  Implementation  of  an  automated  system. 
This  objective  Included  the  following  specific  goals: 

1.  Develop  a  descriptive  model  of  an  Inference  task  that  Is  representative  of  Inference 

tasks  In  the  C2  system  and  would  be  aMnable  to  laboratory  research. 

2.  Determine  appropriate  techniques  for  Masurlng  process  and  perforaance  In  the  Inference 
task. 

3.  Determine  a  prellalnary  set  of  Independent  variables  expected  to  affect  the  Inference 

process  (and  perforaance). 

4.  Develop  a  predictive  aodel  of  the  effects  of  autoaatlon  on  the  operator's  Inference 

processes. 

Accomplishing  these  goals  would  serve  two  purposes:  provide  guidance  to  researchers  at  the  Air 
Force  Huaan  Resources  Laboratory  concerning  variables  of  critical  Interest  In  reltted  field 
research  and  provide  a  franework  for  follow-on  laboratory  research  designed  to  answer  soae  of  the 
questions  outlined  earlier. 


III.  DESCRIPTION  OF  THE  INFERENCE  TASK 

Diagnostic  will  be  defined  as  the  process  of  using  available  cues  to  determine  the 
underlying  or  ’unseen*  cause  of  those  cues.  An  example  Is  Mdlcal  diagnosis  where  the  doctor 
must  Infer  a  disease  that  causes  som  set  of  cues  (symptoms).  If  the  available  cues  are  very 
Inforutlve,  the  Inference  will  be  accurate  and  Mde  with  a  high  degree  of  confidence.  However, 
It  Is  often  the  case  that  the  cues  do  not  convey  enough  Inforutlon,  and  the  Inference  task  takes 
place  amid  psychological  uncertainty. 

In  the  past.  Most  research  addressing  this  type  of  task  assuMd  a  ’single-stage’  process, 
wherein  the  percelver  received  the  cues  and  soMhow  aggregated  or  operated  upon  the  Information 
and  derived  a  judgMnt.  This  was  a  popular  view  for  some  tlM,  partially  because  It  was  amenable 
to  laboratory  experlMntatlon  and  formal  mathoMtlcal  description  and  analysis.4  Two 
approaches  were  common:  The  first  was  to  develop  a  formal  mathematical  model  (such  as  Bayes 
Theorem)  to  specify  optimal  performance  and  then  to  fit  that  model  to  data  obtained  with  human 
subjects;5  the  other  approach  was  to  use  linear  regression  models  to  assess  how  the  subjects 
were  combining  or  utilizing  the  cues  In  generating  the  Inference.5'7  Research  questions 
addressed  In  these  latter  studies  Included  such  topics  as,  what  cues  are  predominantly  utilized 
by  people,7  how  many  dimensions  or  cues  are  used  for  various  tasks  and  whether  these  cues  are 
same  as  In  the  ’real  world,*5  and  whether  experts  cluster  or  weigh  cues  In  a  similar 
fashion. ,,7° 

The  appropriateness  of  these  Mdels  has  been  debated  recently;  therefore,  this  Issue  will  not 
be  discussed  at  length.  However,  two  points  will  be  reviewed.  The  first  Is  the  criticism  that 
most  laboratory  Inference  tasks  Involve  simultaneous  and  orthogonally  manipulated  cues.  This  cue 
Independence  Is  seen  as  being  highly  artificial  and  unrealistic.77  Since  ><umans  develop 
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cognitive  skills  to  deal  with  a  real  and  complex  world.  It  Is  not  surprising  that  they  perforw 
“subopt imally*  on  these  Inference  tasks  where  no  Intercue  correlations  are  preserved. 

The  second  criticise  with  these  approaches  suggests  that  the  Inference  task  should  be  treated 
not  as  a  single-stage  process  but  as  a  Multiple-'  3e  process. ^  This  Is  not  to  say  that  the 
reception  and  the  Integration  of  cues  are  different  stages,  but  that  the  acquisition  of  cues  or 
character istlcs  takes  place  over  time  and  that  this  process  should  be  reflected  In  the 
theoretical  models. 

In  line  with  these  criticisms  and  recent  views  In  cognitive  psychology.  It  will  be  assumed 
that  the  Inference  task  of  Interest  takes  place  In  a  complex  situation  where  the  percelver  must 
sequentially  seek  Information  to  make  the  Inference  Judgment.  In  addition,  that  Information  1$ 
typically  Incomplete  and  varies  In  Its  diagnostlclty.  The  percelver  starts  with  one  or  two  cues 
and  then  searches  for  others  either  to  confirm  hypothesized  causes  or  to  suggest  new  ones.  The 
Inference  process  Is  viewed  as  a  ‘constructive*  process,  much  like  building  a  Jigsaw  puzzle.  One 
does  not  need  all  of  the  pieces  to  be  able  to  Infer  the  nature  of  the  picture;  Instead,  the 
ability  to  draw  the  Inference  will  depend  on  the  combined  Information  provided  by  the  pieces  put 
together. 

In  psychological  terms,  the  percelver  uses  both  Conceptually-Driven  processing  (where  the 
hypothesized  cause  suggests  cues  to  seek)  and  Data-Driven  processing  (where  cues  suggest 
plausible  hypotheses).  The  cyclic  procedure  continues  until  the  percelver  exceeds  some  certainty 
criterion  that  he/she  knows  the  Identity  of  the  cause.  In  some  cases,  a  lack  of  Information  will 
prevent  that  criterion  from  being  reached  at  all. 


IV.  MEASURING  PROCESS  AND  PERFORMANCE 

It  was  suggested  that  the  Inference  Judgment  Is  constructed  over  time  as  Information  is 
acquired.  This  Implies  that  It  Is  Important  to  measure  the  process  by  which  the  percelver  Is 
coming  to  a  conclusion,  as  well  as  to  measure  performance  per  se.  Each  of  these  Issues  will  be 
addressed  In  turn. 

A.  PROCESS 

Several  methodologies  for  measuring  Judgment  or  decision  ‘process*  have  been  suggested. 
Payne1-*  Is  a  predominant  supporter  of  two  of  these  methods  known  as  process  tracing.  The  first 
method  Is  a  class  of  measurement  techniques  where  the  subject's  Information  acquisition  Is 
monitored.  The  subject  must  view  or  select  Information  In  a  way  that  can  easily  be  observed  and 
recorded.  Data  are  obtained  concerning  what  cues  the  subject  samples.  In  what  order,  how  many 
are  sampled,  and  the  amount  of  time  for  the  cue  sampling. 

The  other  method  of  process  tracing  Is  the  collection  of  verbal  protocol.  In  this  technique, 
the  subject  Is  simply  asked  to  ‘think  out  loud*  while  performing  the  task.  Although  this  type  of 
data  can  give  Insight  to  the  subject's  strategies.  It  cannot  be  assumed  that  the  subject  will 
always  verbalize  the  cognitive  processes  as  they  occur. 

After  assessing  the  various  process  measurement  techniques,  a  method  was  decided  upon  which 
seemed  most  suitable  to  an  Inference  task.  The  Inference  Is  actually  a  classification  task, 
where  the  percelver  must  choose  between  class  A,  class  B,  class  C,  and  so  on  (also  possibly  ‘none 
of  the  above*).  The  process  measure  being  suggested  consists  of  two  aspects: 

1.  Allowing  the  subjects  to  acquire  whichever  cues  they  desire  until  they  feel  reasonably 
confident  in  their  choice  (this  Is  similar  to  the  previously  described  Information  acquisition 
measure). 
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2.  Asking  the  subject  after  each  cue  acquisition  to  give  the  hypothesized  cause(s),  along 
with  a  subjective  certainty  rating  (l.e.,  1  «  net  certain  at  all,  7  »  extremely  certain).  An 
example  of  these  measures  will  be  presented  shortly. 

B.  PERFORMANCE 

In  addition  to  studying  cognitive  processes  or  strategies  engaged  In  by  the  percelver,  It  Is 
Informative  to  determine  how  well  the  operator  Is  able  to  Infer  the  cause  of  the  cues.  The  most 
appealing  measures  are  suggested  by  Signal  Detection  Theory  (SDT)  because  It  allows  for  separate 
measurement  of  discrimination  capabilities  and  subjective  bias.14  However,  standard  SDT 
measures  canno*  at  this  time  be  applied  to  more  than  a  two-category  (Signal-Noise)  task.  Swets 
and  Pickett15  discuss  the  problem  of  multiple-category  discrimination  and  suggest  using  Percent 
Correct  as  a  reasonable  solution.  This  Is  Justified  because  the  Inference  task  for  multiple 
causes  Is  actually  conceptually  similar  to  a  forced-choice  task.  It  was  therefore  determined 
that  Percent  Correct  could  be  used  as  a  performance  measure  In  the  diagnostic  Inference  task. 

To  summarize  the  Inference  task  and  associated  measures  of  process  and  performance: 

1.  The  subject  Is  given  an  Initial  cue. 

2.  The  subject  verbalizes  one  or  more  hypotheses,  along  with  a  certainty  rating. 

3.  The  subject  samples  a  new  cue  of  his/her  choice. 

4.  The  subject  verbalizes  revised  hypothesls(es),  along  with  new  certainty  rating. 


(continues  until  select 

An  example  of  data  collected 


exceeds  some  subjective  certainty  criterion  and  chooses  to  stop) 
from  a  subject  Is  given  below: 


Cue  1 
Cue  2 
Cue  3 

Cue  4 


Class  A  Class  B  Neither 

1 

2  2 
4 
7 


It  can  be  seen  that  after  the  first  cue  sampling,  the  subject  hypothesized  class  A  as  the  cause, 
with  a  certainty  rating  of  1.  However,  after  receiving  additional  cues,  the  hypothesized  cause 
was  switched  to  class  B,  with  Increasing  certainty. 


To  determine  performance  (Percent  Correct)  from  the  data,  a  cutoff  point  must  be  chosen  for 
the  certainty  scale.  For  example,  correct  might  be  arbitrarily  defined  as  'more  than  4.*  If  the 
correct  answer  for  the  example  was  class  B,  the  subject  would  be  scored  as  'correct*  on  only  the 
last  observation  given  above.  The  advantage  of  this  method  Is  that  accuracy  of  the  Inference  can 
be  assessed  without  the  subject's  having  to  make  a  strictly  Yes-No  decision.  (This  Is  a  way  of 
•setting*  the  subject's  decision  criterion.)  Kith  this  method  of  data  collection.  It  Is  possible 
to  measure  a  variety  of  process  and  performance  variables  as  the  subject  samples  the  Information: 


1.  Information  sampled. 
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2.  Hypotheses  generated. 

3.  Hypothesis  transition  (at  what  point  they  give  up  and  generate  another). 

4.  Subjective  certainty  criteria. 

5.  Accuracy. 

FEASIBILITY  STUDY 

A  pilot  study  was  conducted  during  the  Summer  Fellowship  period  to  determine  the 
appropriateness  of  the  above  variables.  Eight  subjects  (students  at  Wright  State  University, 
Dayton,  Ohio)  received  course  credit  for  participating  In  the  study. 

Subjects  were  first  given  fictitious  names  of  two  diseases,  along  with  eight  case  studies  for 
each  disease.  The  case  studies  were  described  In  terms  of  patient  initials  and  occupations  and  a 
set  of  symptoms  that  varied  In  number  from  three  to  seven.  There  were  eight  total  symptom 
dimensions  (such  as  blood  pressure,  weight  loss,  etc.).  Four  of  the  dimensions  were  strongly 

associated  with  disease  A,  and  the  other  four  were  strongly  associated  with  disease  B.  However, 
there  was  some  overlap  of  symptoms  across  diseases.  After  subjects  reported  that  they  were 
familiar  with  the  disease  characteristics  (approximately  IS  minutes),  they  were  given  32  new  case 
studies  to  diagnose.  For  each  one,  they  were  presented  with  a  3x5  Index  card  with  initials, 
occupation,  and  one  symptom.  They  were  also  presented  with  seven  other  cards  with  a  symptom 
dimension  (e.g.,  "blood  pressure")  labelled  on  one  side.  They  were  told  to  turn  any  card  over 

that  they  wished,  and  after  each  one,  to  tell  the  observer  their  hypothesis  and  certainty 
rating.  Data  were  collected  In  a  manner  corresponding  to  the  example  shown  previously. 

8ased  on  the  theoretical  considerations  outlined  earlier,  two  effects  were  expected: 

1.  The  certainty  ratings  would  start  low  and  slowly  Increase  until  some  criterion  was 
reached;  at  this  point,  subjects  would  discontinue  Information  acquisition. 

2.  Percent  Correct  would  be  a  positive  function  of  the  number  of  symptoms  sampled  on  a  given 
trial. 

Data  relevart  to  the  first  hypothesis  are  presented  In  Figure  1,  It  can  be  seen  that 
certainty  ratings  did.  In  fact,  start  relatively  low  and  Increase  over  cue  samples;  the  rate  of 
increase  was  highest  for  trials  where  subjects  sampled  only  four  cues,  and  lowest  for  trials 
where  subjects  sampled  eight  cues.  Notice  that  for  trials  where  all  symptoms  were  utilized,  the 

certainty  rating  never  reached  a  level  equal  to  those  for  the  other  trials.  That  Is,  a 

subjective  certainty  criterion  of  approximately  5  was  exceeded  In  all  but  those  trials  where 
subjects  simply  did  not  have  enough  Information  to  generate  an  inference  with  a  high  degree  of 
confidence. 


Figure  1.  Subjective  Certainty  at  ■  Function  of  Cuos  Sampled. 
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This  pattern  In  the  Certainty  data  is  consistent  with  the  Accuracy  (Percent  Correct)  data 
shown  in  Figure  2.  A  liberal  cutoff  was  arbitrarily  chosen  so  that  *3  and  above*  for  the 
Certainty  rating  was  considered  a  "correct*  answer.  Still,  the  scores  were  not  remarkably  high 
(wean  t  correct  for  all  trials  was  .66).  This  would  Indicate  that  subjects  were  not  able  to 
completely  learn  the  correct  structure  of  the  disease-symptom  associations.  Performance  varied 
widely  from  subject  to  subject,  with  a  range  of  .SO  to  .97  for  the  eight  subjects. 


Number  of  Cues  Sampled 


Figure  2.  Accuracy  as  a  Function  of  Cues  Sampled. 

Figure  2  shows  that  subjects'  perceived  uncertainty  had  some  basis,  in  that  performance 
decreased  as  the  subjects  utilized  a  greater  total  number  of  symptoms.  Interestingly,  both  data 
sets  show  the  sane  pattern  of  an  Increase  In  accuracy  and  subjective  certainty  for  trials  where 
seven  cues  ware  sampled.  Additional  studies  will  be  conducted  to  determine  whether  this  effect 
holds  true  for  larger  sets  of  data. 

It  can  be  seen  from  the  data  collected  thus  far  that  the  process  and  performance  measures  are 
appropriate  for  the  task  and  yield  a  rich  variety  of  Information  concerning  the  perceiver's 
strategy.  (It  should  be  noted  that  additional,  more  fine-grained  analyses  are  planned.) 


V.  VARIABLES  AFFECTING  THE  INFERENCE  PROCESS 

Inference  process  and  performance  are  each  affected  by  various  situational  constraints. 
These  Include  characteristics  of  the  percelver,  of  the  task,  and  of  the  situational  environment 
from  moment  to  moment.  A  real-world  Illustration  of  this  complex  situation  will  be  described  as 
a  way  of  introducing  situational  constraints  which  affect  process  and  performance  variables. 

An  Important  Inference  function  within  the  C2  network  is  that  of  the  radar  operator  (and/or 
officer)  who  must  determine  the  Identity  of  aircraft  showing  up  on  the  radar  scope.  This  person 
receives  a  'track*  on  the  scope  and  must  'Infer*  the  identity  of  the  aircraft,  using  auxiliary 
pieces  of  information  or  'cues.'  These  pieces  of  Information  include  Flight  Plan  Data;  Special 
Codes  emitted  by  the  aircraft  (friendly  aircraft);  speed,  heading,  electronic  emissions,  and 
Intelligence  data;  and  possibly,  visual  identification  information.  Some  of  this  information 
will  be  quite  diagnostic  (e.g.,  the  Codes),  while  other  data  will  not  be  particularly  diagnostic 
(speed  or  heading).  Given  enough  time,  the  operator  could  identify  almost  any  track.  If  nothing 
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else,  the  operator  could  send  someone  up  to  took  it  the  aircraft.  The  problem  Is  based  on  the 
fact  that  the  operator  officially  has  a  eaxleue  of  2  minutes  to  Identify  the  aircraft.  In 
addition,  sending  someone  up  for  visual  Identification  very  costly.  In  wartime  conditions, 
the  operator  will  have  to  identify  aany  tracks  In  a  very  snort  period  of  time. 

A  preliminary  list  of  variables  was  developed  that  are  considered  aost  important  In 
determining  the  operator's  performance  (as  defined  by  accuracy).  These  situational  constraints 
on  performance  are  shown  In  Figure  S;  for  example,  *T1ae  Stress*  Is  a  variable  that  will 
negatively  impact  perforaance.  Tiae  Stress  will  be  determined  by  the  nuaber  of  tracks  the 
operator  has  to  identify,  how  far  away  the  aircraft  are  froa  important  locations,  and  the  alert 
status  at  the  time  (i.e.,  white,  yellow,  or  red  alert). 


Figure  3.  Variables  Affecting  Operator  Performance. 


The  saae  variables  are  assuaed  to  also  Impact  the  process  of  generating  an  Inference.  Tiae 
stress  will  result  In  fewer  cues  considered  and  a  lower  subjective  certainty  criterion  (point 
where  the  percelver  1$  willing  to  stop  collecting  data).  Tiae  stress  Is  also  expected  to 
decrease  the  nuaber  of  hypotheses  generated  and  considered  during  the  Inference  process. 

Suspiciousness  of  the  aircraft  travel  will  Increase  the  likelihood  of  an  eneay  aircraft  being 
hypothesized,  resulting  In  cues  being  saapled  which  will  confirm  (or  dlsconflra)  that 
hypothesis.  Dlagnostlclty  or  predictive  validity  of  the  cues  will  result  In  both  a  need  to 
sample  fewer  cues  and  a  higher  subjective  certainty  concerning  the  Inference.  Finally,  operator 
experience  is  assuaed  to  enhance  the  Inference  performance,  but  It  Is  not  totally  clear  how  that 
variable  will  lapact  the  process. 


VI.  IMPACT  OF  AUTOMATION 

Before  discussing  the  lapact  of  Implementing  a  computer-aiding  systea.  It  Is  necessary  to 
define  the  nature  of  the  autoaated  systea.  Several  researchers  have  developed  taxonomies  of 
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types  of  automation;16  Most  are  admittedly  iMprecIse.  For  the  present  purposes,  a  continuum 
will  be  assuMd,  with  a  completely  Manual  method  of  task  accomplishment  at  one  end  and  complete 
automation  at  the  other  end  of  the  continuum.  An  example  of  this  type  of  continuum  Is  shown  In 
Figure  4.  One  of  the  more  prevalent  types  of  automation  Is  given  on  the  top  right  of  the  scale; 
that  Is,  the  computer  completely  performs  the  task  and  provides  an  "answer"  to  the  operator,  who 
then  decides  If  he/she  wants  to  believe  and  utilize  that  answer.  This  type  of  automation  Is 
currently  being  planned  for  the  radar  Identification  task  described  earlier. 


F 


Manual; 
Operator  Does 
All  the  Work 


Computer  Only 
Provides 
Information  to 
Operator  In  Some 
Efficient  Form 


Computer  Performs 
Task  Until  Fads, 
Then  Operator 
Attempts  Task 


Computer  Completely 
Performs  Task  and 
Operator  Decides 
Whether  to  "Accept" 
Answer 


i 


Computer  Suggests 
Most  likely  Answers 
or  Hypotheses: 
Operator  Does  the 
Rest 


i 

Computer  Analytically 
Computes  Answer 
Based  On  Input 
From  Operator 


Automated: 
Computer  Does 
All  the  Work 


Figure  4.  Degrees  of  Automation. 


A  predictive  model  can  be  outlined  at  this  point,  describing  the  Impact  of  1mp)~ment1ng  this 
type  of  automation.  First,  we  can  say  that  most  of  the  variables  expect:u  to  affect  the  human 
percelver  will  probably  not  affect  the  performance  of  the  autometcu  system.  Thus,  under 
conditions  of  stress,  the  human  parformance  wilt  deteriorate,  whereas  machine  performance  will 
not.  The  only  variables  expected  to  affect  machine  performance  will  be  the  characteristics  of 
the  cues  themselves  (s»?  Figure  5).  This  puts  the  operator  In  the  place  of  deciding  whether  to 
"trust"  the  machine,  knowing  that  the  machine  can  perform  the  task  more  quickly  and  objectively 
In  times  of  duress.  The  operator's  decision  to  uso  tho  answer  provided  by  the  system  will  depend 
on  how  much  time  he/she  has,  the  seriousness  of  the  consequences,  and  the  nature  of  the 
Information  (e.g..  If  It  Is  a  suspicious  activity).  In  addition,  the  operator's  decision  to  use 
the  answer  provided  will  strongly  depend  on  tho  operator's  feelings  about  his/her  own  ability 
versus  the  machine's  history  of  reliability  end  accuracy.  If  the  machine  has  a  relatively  low 
"hit  rate,"  the  operator  will  be  more  Inclined  to  consider  Its  answer  worthless  and  go  mostly  on 
his/her  own  Judgment. 


An  Interesting  point  mhlch  should  bo  considered  is  the  possibility  that  tho  operator  might 
treat  the  answer  provided  by  the  machine  as  merely  "another  cue."  Rather  than  determining 
his/her  Inference  based  on  tho  standard  set  of  cuts  and  than  comparing  this  with  the  automated 
choice,  the  operator  may  simply  uso  that  choice  during  tha  Inference  process,  treating  It  as 
he/she  does  all  of  tho  other  pieces  of  Information. 
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Figure  5.  lap  1  mentation  of  Autonation  In  Inference  Task. 


VII.  RECOMMENDATIONS 

The  previous  section  outlined  hypothesized  effects  of  situational  constraints  on  Inference 
process  and  performance.  In  addition,  s  preliminary  predictive  nodel  mas  constructed  relating 
the  Implementation  of  an  automated  subsystem  within  a  'manual*  Inference  system.  The 
relationships  between  the  variables  need  to  be  obtained  empirically  to  confirm  the  theoretical 

assumptions. 

I  propose  to  do  this  by  developing  a  laboratory  Inference  task  that  has  the  characteristics 
outlined  earlier  (complex,  sequential  cue  acquisition,  etc.).  This  Inference  task  will  be  run  on 
a  computer  terminal  so  that  the  subject  can  request  cues  and  respond  with  hypotheses  and 
certainty  ratings.  This  will  allow  collection  of  data  relevant  to  the  process  and  performance 
variables  outlined  previously. 

For  assessment  of  the  processes  used  In  a  manually  performed  Inference  task,  situational 
constraints  will  be  manipulated  and  their  effects  on  process  and  performance  variables  measured. 
This  will  allow  determination  of  what  processes  are  utilized  under  various  task  conditions. 

A  second  phase  of  the  follow-on  research  will  be  to  provide  the  subject  with  a  computer  aid. 
This  will  be  a  simulated  computer-aiding  system  because  the  answer  to  be  given  to  the  subject 
will  be  predetermined  by  the  Experimenter.  In  this  way,  the  accuracy  of  the  answers  provided  to 
the  percelver  can  be  manipulated  as  an  Independent  variable.  It  Is  expected  that  as  accuracy  of 
the  automated  system  Increases,  the  use  of  the  machine  will  also  Increase  In  an  exponential 
fashion.  Situational  constraints  will  also  be  varied,  similar  to  those  used  In  assessment  of  the 
completely  manual  Inference  process. 
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